* 


AD-A264  802  ENTATION  PAGE 


M 1 1 


Form  Approved 
OMBNo.  0704-0183 


aS9  1  hour  pet  rsaponss.  Including  me  lime  tot  tovieAtcj  inciiucticns.  jcaicting  cjuains  diia  '-ouicei.  gameiir  g  and 
iformaiJon  Serid comments  regarding  ihJs  burden esr-maie  cr  any  cir.et  aspect  cf  coiJecikjn  ir'crmaiien.  m c:  jCing 
arectoraleforlnfcrmaitonOperailons  and  Reports.  12t5jeftef'scn  Davis  highway.  Su?t©t204.  Aningion  VA  222C2  43C2. 
(0704-01881.  Washinatoo.  DC  20503 


1  AGENCY  USE  ONLY  (leave  bfeinAJ 


2.  REPORT  DATE 

March  1993 


12a.  DISTRIBUTiOWAVAJLASIUTY  STATEMENT 


Approved  for  public  release;  distribution  is  unlimited. 


13.  ABSTRACT  (Maximjm  200  words) 


3  RL'POHTTTPE  A.''j0DA7ESCCVERf;O 

Professiuna)  Paper 


4.  ■nTLEANDSUamE 

5  FUNDir^  NUMBERS 

EVOLVING  NEURAL  NETWORK  ARCHITECTURE 

In  Hou.se  Funding 

6  AUTHORtS) 

J.  R.  McDonnell,  D.  Waagen 

r  PERFORMING  ORGANIZATION  NAME<S)  AND  ADDRESS(ES) 

Naval  Command,  Control  and  Ocean  Surveillance  Center  (NCCOSC)  i 

RDT&E  Division 

San  Diego,  CA  92152-5001 

3  PERFORMING  CRGANtZATCN 

PEPORT  number 

1  9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

i 

Naval  Command,  Control  and  Ocean  Surveillance  Center  (NCCOSC) 

RDT&E  Division 

San  Diego.  CA  92152-5001 

10  SPCNSCfliNG.VONfrORlNG 

AGENCY  REPORT  NUMBER 

1 1 .  SUPPLEMENTARY  NOTES 

n-riC 

This  work  investigates  the  application  of  a  stochastic  search  technique,  evolutionary  programming,  for  developing  self- 
organizing  neural  networks.  The  chosen  stochastic  search  method  is  capable  of  simultaneously  evolving  both  network  archi¬ 
tecture  and  weights.  The  number  of  synapses  and  neurons  are  incorporated  into  an  objective  function  so  that  network 
parameter  optimization  is  done  with  respect  to  computational  costs  as  well  as  mean  pattern  error.  Experiments  are  con¬ 
ducted  using  feedforward  networks  for  simple  binajt7  mapping  problems. 


is 


J  o 


Pubiishod  in  Proceedings,  SPIE  92  International  Symposium  Neur.al  and  Stochastic  Methods  in  Image  and  Signal 
Processing,  Vol  1766. 


u  suajEcr  terms 

NUMBER  OP  PAGfS 

neural  network.s 

mean  pattern  error 

stochastic  search  technique 

18  PRICE  CODE 

'';:i<<frYCiAS.S;f  iCAr'C.N 

0*  REf'ORr 

ifj  •  >  f  Ifnrv  rj  A SSIF  JCA  MON 
c,i 

10  <.i  r.1  lOiTv  rs  A  .  .  >  c  AtH.-N 

OF  ALLS^RACT 

?G  'JMnAUCNCT  Ai?:'.,U(AC'i 

UNCI.ASSIFiED 

liN(;LA.SvSIFIKD 

UNCLASSIFIF.D 

SAME  /\S  RIcPORT 

Evolving  neural  network  architecture 


John  R.  McDonnell  and  Don  Waa,i;en 

NCCOSC,  RDT&E  Div. 

San  Die^o.  CA  92152-5000 

ABSTRACT 

This  work  investigates  the  application  of  a  stochastic  search  technique,  evolutionary-  progra/nmini; ,  for  daclopin-.;  sclj- 
organizing  neural  networks.  The  chosen  stochastic  search  method  is  capable  <>f  sim-altaneously  e-t-ol  .ing  both  network 
architecture  and  weights.  The  number  of  .synapses  and  neurons  are  incorporated  into  an  ohjcciiva  function  so  that  network 
parameter  optimization  is  done  with  respect  to  computational  co.sts  as  well  as  mean  pattern  error.  Experiments  arc  conducted 
using  feedforward  networks  for  simple  binary  mapping  problems. 

1.  INTRODUCTION 

The  neural  network  design  process  is  largely  heuristic.  The  designer’s  previous  expenence  (or  the  work  ot  other  resettrehers  i 
oftentimes  dictates  an  initial  network  configuration  for  the  problem  at  hand,  i’  the  network  can  he  trained  to  achiese  ihe 
designer's  goals,  then  the  design  process  is  terminated.  If  succe.ss  is  not  attained,  then  a  te.sting  phase  which  is  largely  trial 
and  error  ensues.  The  end  result  can  often  be  ;  network  with  excess  parameters  and  little  regard  for  computational  costs. 

This  work  attempts  to  incorporate  the  computational  costs  associated  with  neural  network  configurations  into  the  optimization 
procedure.  Potential  benefits  of  a  network  which  has  an  optimized  architecture  include  increased  throughput  for  real-time 
signal  processing  applications  as  well  a.s  decrea'^d  memory  requirements.  Determining  both  network  parameter^  and 
structure  simultaneously  requires  a  .search  procedure  which  is  amenable  to  combinatorial  optimization  tasks.  The  more 
successful  algorithms  for  these  types  of  problems  have  generally  been  stiKhastie  .such  as  simulated  annealing,  genetic 
algorithms  and  simulated  evolution.  The  simulated  evolution',  or  evolutionary  programming  (HP),  paradigm  has  been  show  n 
to  have  the  desired  attributes:  combinatorial  optimization  capabilitie.s-,  the  ability  to  train  neural  networks'  a.^  well  as 
determine  model  structure'*. 

Many  investigations  have  been  undertaken  m  an  effort  to  obtain  desired  I/O  mappings  with  variable  configuration  networks. 
Ash^  developed  the  dynamic  node  creation  (DNC)  algorithm  which  created  new  nodes  in  the  hidden  layer  when  the  training 
error  rate  fell  below  an  arbitrarily  chosen  cntical  value.  Hirose  et  al.‘‘  use  the  same  approach  for  node  creation,  but  akso 
remove  nodes  when  small  error  values  are  attained.  Dow  and  Sielsma^'*  have  developed  post-proce,ssing  algonthras  which 
remove  or  prune  excess  nodes  from  network  hidden  layers.  These  algorithms  are  implemented  a.s  a  two  stage  process.  The 
first  stage  removes  nodes  W'hich  have  constant  output.',  or  are  coniplinientai y  to  other  nodes.  The  .secoiul  st.ige  len'ioves  noJe^ 
that  are  independent  ot  the  other  node,s  within  the  layer  and  who.se  output  is  not  required  to  achieve  a  solution 

Li’  has  developed  a  generalization  of  the  baekpropagation  algorithm  which  minimizes  both  the  total  syeleiu  enoi  and  the 
norm  of  the  wcight.s.  In  .simulation,  this  approach  yielded  null  portions  of  the  weight  .space  providing  an  implieit  means  ot 
network  optimization  (weights  and  structure).  .Simulation  results  also  resulted  in  nearly  invariant  weight  sets  regardless  of 
the  initial  weight  values  whereas  straight  baekpropagation  did  not,  Bailey**’  generates  neural  network  structure  by  evaluating 
synapse  importance  and  system  error  (synap.se  el fe<  liveness)  criteria.  If  Inith  synapse,  effectiveness  and  iinportanee  become 
low  then  It  becomes  eligible  tor  removal.  I'enorio  and  1  ee"  have  applied  simulated  annealing  to  a  group  methiHl  d.it.i 
handling  (GMDH)  scheme  to  obtain  a  .self  organizing  neural  network  (SONN)  \  hierarchical  minimum  dc.scription  Icngih 
(MDL)  function  called  the  structure  estimation  criteria  is  used  as  the  ob|ective,  lunelion.  The  resulting  network  is  .sp;iisi  lv 
connected  as  each  node  is  arbitrarily  limited  to  two  inputs.  Sinec  a  low  order  Kolmogorov  (labor  pi'lyr.uniial  apptoxim.itum 
IS  implemented]  for  the  node  aclivalion  function,  more  wcight.s  than  connections  e.iist  ileitz  ei  a!  give  a  cotu  isc  ovei\u  e>. 
of  other  work  on  neural  nelwotk  (onstru.  Imn  and  ('iiininc  alconihins. 


The  premise  of  this  work  is  that  the  designer  can  postulate  an 
objective  function  which,  if  optimizerl,  will  yield  desirable  results 
for  the  task  at  hand.  Further,  the  approach  u.sed  in  this  study  seeks 
to  take  advantage  of  computational  resources  during  the 
design/training  phase  thereby  removing  the  burden  of  evaluation  by 
trial-and-error  from  the  de.signer.  For  purposes  of  discu.ssion.  Fig. 

1  illustrates  the  structure  of  a  hypothetically  evolved  neural  network. 

The  connectivity  and  number  of  nodes  in  the  hidden  layer  are 
determined  via  a  multi-agent  stochastic  search  technique,  'fhe 
application  of  EP  to  this  task  is  investigated  in  three  distinct  phases. 

Initially,  the  EP  training  algorithm  for  neural  networks  given  in  the 
next  sectioifis  generalized  so  that  connectivity  between  nodes  can  be 
randomly  selected.  In  the  next  phase  of  the  study,  the  number  of 
hidden  units  is  ?  random  variable  which  is  incorporated  into  the 
stochastic  search.  The  final  phase  integrates  methods  employed  in 
the  previous  phases  into  a  single  EP  search  algorithm  to  achieve  a 
network  such  as  that  showm  in  Fig.  1. 

The  evolutionary  programming  paradigm  is  outlined  in  the  next 
section  along  with  applications  of  the  EP  search  technique  to  fully- 
connected  feedforward  neural  networks.  Subsequent  sections 
investigate  evolving  the  connectivity  structure  of  the  network  and  the 
number  of  nodes  in  the  hidden  layer(s).  These  method.s  are  then 
combined  to  yield  self-organizing  neural  networks. 

2.  APPLYING  EVOLUTIONARY  PROGRAMMING  TO  NEURAL  NETS 

2.1.  Evolutionary  Programming 

Evolutionary  programming  is  a  neo-Darwinian  search  paradigm  first  sugge.sted  by  Fogel  n  n/. ’  I'hi.s  sto.hastk  seai,h 
method  is  typically  applied  as  a  global  optimizer.  EP  has  been  successfully  applied  to  a  variety  ot  ('ptimizalion  problems 
including  the  traveling  .salesman  problem,  parameter  estimation  and  system  ID,  and  neural  net  training. 

The  EP  optimization  algorithm  for  IcKating  extrema  of  any  general  function  can  be  described  by  the  (ol/owmg  steps 

!.  Form  an  initial  population  (x)  of  size  2N.  The  parameters  x  associated  with  parent  element  P,  are  rnndomh 
initialized  from  a  user  specified  search  domain. 

2.  Assign  a  fitness  score  Sfx)  to  each  element  P,(x)  in  the  population. 

3.  Reorder  the  population  ha%ed  on  the  number  of  wh/Lv  generated  from  a  siorhastir  competiii<>n  pnu-r.^\. 

4.  Generate  offspring  ....  Pth.,)  of  the  highest  ranked  N  elements  (P„  ....  ,)  in  lh<  /hipidtHion. 

.“i.  lamp  to  step  2. 

The  generality  of  this  algorithm  lend.s  power  to  it.s  implementation  besides  providiiu;  a  svsteinatu  im  ans  <  q  sten  hastu  si-aieh 
the  u.ser  i.s  not  bound  to  any  particular  coding  .structure  nor  mutation  strategy.  .‘\s  previously  stated.  M’  is  uso,i  m  tins 
investigation  since  it  is  well  .suited  for  smuiltaneously  evolving  niiKlel  stnietiire  and  parameteis 

2.2.  Training  Neural  Networks  with  EP 
2.2.1.  Determining  Network  Weights 

Since  t.P  IS  a  general  purpose  optimization  procedure,  it  can  in-  reaiiily  useil  tor  lU  temiinuu’  n  uial  netwnik  omiei  tion 
strenglh.s.  Ilic  objective  lunction  of  the  l-,P  training  approach  (ot  tiilly  (.(inuected  teedtni\',.,iil  m  luoikv  r-,  the  s.iui,-  a--  that 
used  in  hackpiopagalion:  minimize,  the.  sum  sqiiaied  erioi  tuiu  lioii  P  a,l  A  .omns'u  iiu  lii.  o  lo  noim.ili/e 


Figure  /,  .4  hypothetically  evolved  network 

structure  with  variable  hidden  units  and 
connectivity. 


E  by  the  number  of  training  patterns  presented.  'Fhis  section  applies  the  algorithm  given  in  the  previou.s  section  to  ritural 
networks  and  then  shows  results  for  sample  training  runs  using  simple  binary  mappings.  It  should  be  noted  tha*  the  examples 
given  are  by  no  means  statistically  significant  and  are  primarily  shown  for  exemplary  puqxises. 

Each  fully-connected  feedforward  network  in  the  population  P  of  networks  may  be  repre.scntcd  by  the  weight  array 

=  w[i] [layer} lfrom_node][iojiodeJ 

For  this  application,  the  initial  population  was  instantiated  with  weights  chosen  from  a  Ul-0.5,  0.5}  distribution.  Next,  a 
cost  is  assigned  to  each  network  in  the  configuration.  The  N  members  of  the  population  generate  offspring  (perturbed  weight 
sets)  according  the  equation 

IV,  =  VV  +  6W^ 

where  typically  t  N(0,  Sf-E)  with  a  scaling  coefficient  Sf.  The  scaling  factor  is  a  probabilistic  analog  to  the  slepsize 
used  in  gradient  descent  methods  and  may  also  be  treated  as  a  random  variable  within  the  EP  search  strategy.  The  variance 
of  the  weight  perturbations  is  bound  by  the  total  system  error  in  this  application.  To  emulate  the  probabilistic  nature  of 
survival,  a  pairwise  competition  is  held  where  individual  elements  compete  against  randomly  chosen  members  of  the 
population.  For  example,  if  network  is  randomly  selected  to  compete  against  network  4’.,  then  a  win  is  awarded  to 
network  iff,  <  Ej.  Another  approach  would  be  to  let  the  probability  of  network  <i>j  winning  be  E/(E/+EJ.  The  N 
networks  with  the  most  wins  are  then  used  to  generate  offspring  and  the  process  is  repeated. 

Figures  2-4  shows  the  effect  of  varying  different  parameters  in  training  a  2-2  /  neural  net  for  the  XOR  mapping,  log,  2 
shows  the  best  mean  sum-squared  pattern  error  for  various  scaling  factors  from  a  population  with  10  parent  networks.  As 
shown,  modifying  the  scaling  factor  effects  the  convergence  rate  of  the  training.  The  smaller  scaling  factors  fend  to  produce 
smoother  curves  (smaller  changes)  while  the  larger  scaling  factors  yield  more  dramatic  results  due  to  larger  step  sizes.  The 
number  of  networks  over  which  the  search  is  conducted  is  generally  user  defined.  However,  the  population  size  need  not 
be  a  static  parameter  as  it  can  increase  or  decrease  over  the  generations  of  the  search.  Training  result.s  with  a  5x  increa.se 
in  population  size  is  shown  in  Fig.  3  for  a  scaling  factor  of  5,  =  JOO.  Fig.  4  shows  successful  training  is  po.ssiblc  with  a 
randomly  chosen  scaling  factor  for  5^  «  UfOJOOOJ  and  N  =  10.  The  training  run  shown  did  not  converge  as  fast  as  other 
runs,  but  it  did  produce  a  smoother  scaling  factor  sequence. 


f'igure  2.  hP  training  of  a  2-2- !  XOK  mapping  network  for  varions  'scaling  factors,  S  It) 


Figure  J,  EP  training  of  XOR  mapping  network  Figure  4.  EP  training  of  XOR  mapping  network  with 

with  different  population  sizes,  Sf  =  WO.  variable  5,. 

2.2.2.  Hard-limited  Acti?ations 

Beside.s  yielding  only  locally  optimal  solutions,  it  is  necessary  that  activation  functions  be  continuously  differentiable  when 
gradient  descent  methods  are  employed  for  training  multi-layer  perceptrons  (MLPsV  The  continuity  condition  is  not  a 
rec(Uirement  when  stochastic  methods  such  as  simulated  anneallng'^  genetic  algorithms"’  or  EP  are  used.  An  interesting 
method  has  been  developed  by  Winter  and  Widrow'^  for  training  MLPs  with  bipolar  activation  functions.  This  method, 
termed  the  Madaline  Rule  11,  consists  of  applying  the  minimal  disturbance  principle  to  weights  assrxiiated  with  individual 
■ADALINES.  If  better  re.sults  are  obtained,  then  the  new  weight  values  ih.  kept;  otherwi.se.  the  new  weights  are  ignored. 
If  the  training  process  exhausts  tnals  involving  a  single  ADALINE,  pairwise  (or  higher)  adaptations  are  attempted. 

The  3-bit  parity  problem  has  been  chosen  as  an  example  for  training  MLPs  with  hard-limited  activation  functions  using  EP. 
Fig.  5  compares  training  results  when  sigmoidal  and  binary  activation  functions  and  are  used  in  a  J-J-J  network.  As  shown, 
the  discrete  system  error  states  which  result  when  using  a  binary  activation  function  are  quite  pronounced.  The  concept  of 
a  different  activation  function  for  different  neurons  can  also  be  ea,vily  employed  within  an  EP  search.  Each  different 
activation  function  could  be  treated  a.s  a  random  variable,  and  he  di.scretely  indexed  within  a  neuron  structure. 

2.2.3.  Combining  Backpropagation  and  EP 

It  IS  po.s.sible  to  do  a  local  and  global  .search  in  parallel.  Such  a  strategy  ha.s  been  inve.stigated  by  Waagen  c/  in'."'  m  the 
development  of  the  Stcxihastic  Direction  Set  (SDS)  method.  This  method  mainlain.s  the  integrity  ol  .searching  tor  a  solution 
set  of  parameters  without  explicitly  requiring  the  analytical  function  or  empirically  determined  derivatives.  T3ie  SDS  method 
u.ses  an  augmented  (with  orthogonal  decomposition)  1  looke-Jecves  technique  for  the  local  search  while  EP  is  used  for  finding 
the  global  solution  neighborhix^d.  Tlie  variance  of  the  stochastic  step  sire  i.s  held  constant  over  the  duration  of  the  search 
I  Wo  offspring  are  generated  for  each  ha.se  point;  one  locally  and  one  globally.  A  Itxal  offspring  replaces  its  parent  if  it 
achieves  a  better  filne.s,s  score,  EP  and  backpropagation  training  can  be  implcniented  together  in  a  .similar  parallel  fashion 
Fig.  6  shows  a  .serial  implementation  wherein  a  J-J  l  network  is  first  trained  with  EP  and  then  rclined  using  backpropagation 
for  the  3  bit  parity  mapping.  Tins  sample  run  wa.s  generated  with  10  parent  nciwork.s  and  a  scaling  factor  .V,  .f.f  I  he 
backpropagation  parameters  inciudtxl  a  stepsi/x;  of  rj  ^  0.'.>aruf  a  moiuenluni  cocffitienl  ol  <i  0  2 


1 


CENER»TtOV/tPOC« 


J 


Figure  5.  EP  training  for  3-bit  parity  mapping  with  Figure  6.  A  serial  EP-BP  training  session  for  the  3-bii 
both  sigmoidal  and  binary  activation  functions.  parity  mapping. 


3.  EVOLVING  CONNECTIVITY 

This  portion  of  the  investigation  addresses  reducing  the  number  of  synapses  between  the  neurons.  Typically,  work  in  this 
area  incorporates  a  function  of  the  weight  magnitudes  in  the  objective  function.  Hertz  ei  al.  for  example,  di.scu.s.s 
optimizing  an  objective  function  J  described  by 


J 


V 


vv. 


where  E  is  the  mean  sum-squared  pattern  error.  Weigend  et  al.'^  implement  a  similar  modification  with  the  weights 
normalized  by  an  arbitrary  parameter 


J  =  E 

u 


1  4 


This  study  investigates  modifying  the  object  function  according  to 


J  ^  aE  ^  UN 


ronnrcttons 


whereby  the  cost  a.s.soctated  with  the  number  of  connections,  between  neuron.s  is  scaled  and  incorporated  into  the 

cost  function,  f  or  the.se  studies  a  value  of fi  =  0.0001  was  u.sed  with  ct  =  I .  A  heuristic  which  might  be  employed  would 
be  to  let^  ~  aE^^/N^  thereby  incorporating  the  desired  training  error  and  the  maxitnum  numter  of  connections  pos.sible 
over  the  specified  domain  to  rea-sonably  weight  the  co.st  as.sociated  with  the  number  of  connection.s. 


Figure  7.  Evolving  connectivity  for  the  XOR  mapping.  Figure  S.  The  final  evolved  connectivity  for  the  XOR 
The  final  architecture  is  shown  to  the  right.  mapping  network. 


To  implement  this  capability,  a  connectivity  array  has  been  utilized  where  c[i][layerJ[fromjiode][to_node}  -  7  if  a 
connection  exists  and  0  if  no  connection  is  present.  At  the  start  of  the  run,  the  connectivity  array  is  set  to  1  giving  a  fully- 
connected  feedforward  network.  The  designer  must  specify  the  number  of  neurons  over  which  the  search  is  conducted.  This 
determines  the  maximum  number  of  connections.  From  the  range  of  connections,  a  synapse  is  chosen  at  random  and 
modified  based  on  its  current  state.  It  is  connected  if  it  has  been  disconnected  or  disconnected  if  it  is  currently  connected. 
Essentially,  the  bit  is  flipped.  The  number  of  connections  which  may  be  affected  at  each  mutation  is  arbitrarily  set  by  the 
designer  or  detemuned  in  a  random  fashion.  The  connection  array  is  incorporated  in  the  neuron  input  dot  product  term 
thereby  nulling  any  signals  between  disconnected  neurons  As  before,  werights  are  continually  modified  in  the  event  that  a 
neuron  pair  is  reconnected. 

The  results  of  evolving  connectivity  using  a  network  with  8  hidden  units  for  the  XOR  mapping  is  shown  in  Figures  7  and 
8.  After  1000  generations,  the  network  with  the  lowest  cost  function  has  12  connections  as  shown  in  Fig.  8.  Although  the 
structure  can  be  reduced  to  replicate  a  fully-connected  2-2-1  feedforward  architecture,  it  must  be  done  through  post¬ 
processing. 


In  an  effort  to  place  greater  emphasis  on  signal  propagation  through  the  network,  an  alternative  strategy  has  been  developed 
though  not  yet  implemented.  This  strategy  assigns  a  probability  of  cormection  from  neuron  k  in  layer  j,  to  subsequent 
layers  contingent  on  inputs  to  all  the  neurons  in  level  j 


This  technique  assume,s  a  random  connection  process  exi.sls  betwwn  the  input  layer  and  lirst  hidden  layer.  Pixir  suiiiiion'i 
(such  a.s  exce.ss  emphasis  on  a  single  node)  will  not  be  su.stamed  unle.ss  the  fitness  value  is  competitive  with  other  memher 
of  (he  population. 


4.  EVOLVING  HIDDEN  NODES 

The  standard  backpropagation  approach  to  training  neural  networks  entails  specifying  the  nuinhct  of  layers  and  the  number 
of  nodes  in  each  layer.  This  portion  of  the  investigation  allows  (he  number  of  nrxlcs  in  tlie  huiden  iayer(s)  tej  he  ireated  a.s 
a  random  variable.  The  EP  training  technique  given  in  section  2.2.1  is  still  u.sed  although  other  training  methods  (sueh  as 
backpropagation)  may  be  applicable. 

The  method  outlined  in  section  2.2.1  is  slightly  modified  to  allow  for  a  variable  number  ol  hidden  nodes;.  The  nu.xiinum 
number  of  hidden  nodes  specified  in  the  program  is  the  domain  over  which  the  search  is  conducted.  Thus  tuts  the  obvious 
shortcoming  that  it  is  still  possible  to  under  estimate  the  number  of  node,s  which  may  be  required  to  solve  i  partu  ular 
problem.  This  portion  of  the  investigation  maintains  a  fully-connected  feed-forward  areluiecture,  Tfic  cost  funcSu  ii  used 
is  given  by'' 


y  =  OtE  ^ 

where  a  =  1  and  y  =0.01.  A  .scaling  factor  of  7— 0.01  allows  smaller  networks  with  larger  mean  .sum-srjuared  pattern  error 
terms  to  be  selected.  At  each  weight  perturbation,  the  number  of  hidden  nodes  is  randomly  selected  from  a  u.ser  specified 
domain. 

Fig.  9  shows  the  re.su!ts  for  the  XOR  mapping  problem  with  varying  hidden  units.  The  maxmuirn  nurnlxT  of  hidrlen  iinils 
possible  was  arbitrarily  set  at  20.  After  200  generations  the  network  has  evolved  a  2-2-/  .stnicture  a.s  well  as  an  appropriate 
set  of  weight  parameters.  At  the  end  of  ,500  generations  a  mean  sum-.squared  pallein  err.T  of  0  OOOOt  iv  attained  !ir  Pi 
shows  the  results  for  the  XOR  mapping  problem  if  the  search  domain  is  increased  to  .SO  hidden  iinit.s.  In  this  case,  a  2-2-1 
structure  is  also  evolved  simultaneously  with  the  weight  parameters.  At  500  generations  a  mean  sum  squared  pattern  error 
of  0.0001  is  achieved.  Both  sets  of  experiments  were  conducted  with  Sf  =  100  and  10  parents. 
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figure  9.  Evolving  hidden  units  for  the  XOR  mapping.  figure  10.  Evolving  hidden  itnits  for  the  XOR  mapping. 
The  search  H’a.s  conducted  over  20  possible  hidden  units,  llie  search  was  conducted  over  50  possible  hidden  units. 


5.  EVOLVIN(;  STRUC  riiRK;  A  SK1,K-{)R(;AN1ZIN(.  NK  IAVORK 


A  nicety  of  using  the  lil'  sc^irch  technique  is  that  additional  ternis  can  lie  incorporated  m  tlsc  ob|e..tise  tunctu  u  vothciut 
requiring  a  major  prograniniing  niodii'ication.  An  enampie  ot  this  feature  is  the  iiiulti  JinK  risiional  path  planning  ■.‘..ak  d.-ue 
hy  Page  et  al.''* .  Wah  and  Knplani''*  have  incorporated  training  time  into  their  ol  jectiv  e  tun.  tion 

J  A  H  X  Traiiiin}^  Tinu'  C  x  Cost 


where  A,  R  ;md  C  are  constants.  The  Cost  lenn  is  a  (unction  ol  the  netwiirk  si/e  anti  niiinlvr  o)  weigiit.':  and  o  I’lsen  h  , 


Cost 


N 


o.\p. 


An  inve.stigation  very  similar  to  this  work  ha.s  been  conducted  by  Bornholdt  and  Graiiden/.’"  major  diflerence  betwci  n 
the  methods  is  that  Bornholdt  and  Graudenz  u.sed  genetic  algorithncs  for  cvolv.iig  the  r.;,:v.ork  .stnicture.  i'urthei,  liie 
networks  implemented  in  the  referenced  work  can  have  asynchronous  update.s  and  recurrent  structure  in  the  hidden  layer 
Bornholdt  and  Graudenz  define  a  dilution  ratio  D  =  .  for  e.xample,  the  dilution  ratio  for  the  network  shown 

in  Fig.  8  is  D  =  0.33.  This  re.sult  does  not  have  the  small  dilution  ratios  found  using  recurrent  hidden  structure  l-Tr  the 
simple  XOR  mapping,  it  does  not  take  as  many  generations  to  train  since  the  stability  issues  associated  with  re.. iirrerit 
structure  does  not  effect  the  system 

Currently,  the  feed-forward  structure  is  maintained  and  the  objective  liinction  i.  niodiii -d  to  m, ,  ap.  eat;  b.  tli  ...;  .  ■  •  .u  -.  j 

with  the  number  of  comiection.s  and  number  of  neurons 

J  =  OiE  t  BN  +  yN  , 

^  ccnnections  •  nodfs 

Initially,  the  weighting  coefficients  were  kept  the  same:  a--  t .  p  -0.0001 ,  y  -  0.01.  I'he  methc'ds  useri  in  sections  1  anil  4 
were  combined  and  tested  on  the  XOR  mapping.  Fig.  ( !  shows  (he  framing  re.sults  (or  ,i  liuiji-n  l.ivcr  do/iuin  ot  .su  luJJ.-;! 
units.  After  25CK)  generations,  the  network  ha.s  evolved  to  a  2-2-1  structure  llowcw.-i,  the  mean  'iinv^quared  jialt.-'in  ciio: 
has  not  been  sufficiently  reduced  as  shown  m  Fig.  1  1(b).  Tins  is  due  to  the  inannei  m  w  in  ,  li  ,>1)  pnng  arc  itcnciatcd  !.a  i: 
new  oftspring  contains  a  new  structure  as  well  as  perturbed  weight  set.  Ghaicio  .ue,  ■iciv  little  v-eiitht  adaj'i.ition  i. 
accomplished  for  any  given  structure. 

The  adaptation  problem  became  moie  apparent  when  the  same  apjiroach  wss  taken  wnh  the  t  parity  mapping  fi 
overcome  thi.s  limitation,  two  offspring  are  generated  for  each  parent  The  first  otlspring  is  generated  with  a  pcrtuibed 
weight  .set  and  new  structure.  The  second  offspring  is  generated  with  a  perturbed  weight  set  and  the  same  stnictiiie  as  its 
parent.  Results  for  the  3-bit  parity  mapping  problem  are  shown  in  Fig.  12,  This  experiment  was  conducted  with  weighting 
coefficients  of  a  =  1 ,  p  — 0.0001 ,  y=0.00I .  The  hidden  layer  domain  consi.sted  of  2.S  hidden  units.  After  500  generations, 
the  mean  sum-squared  pattern  error  has  been  sufficiently  reduced  (or  II  hidden  units  as  .hewn  m  Fig  I2(bi  Fiutli  i 
optimization  occurs  by  reducing  the  number  of  connections  as  shown  in  I  ig  l.’(c).  I  he  tin.il  i  vmliguiation  is  shown  m  i  ic 
13.  Obviously,  not  all  of  the  remaining  hidden  units  are  connected  and  the  bias  icdundancv  c.tn  be  clmimalcd  imilv  i 
reduce  the  node  count.  The  statistics  at  the  end  of  ,5000  generations  are  I  S,  1 1 .  C  0.()O0'.C2 .  J 

0.012852  and  a  dilution  ratio  of  D  0  15. 

This  modified  approach  was  applied  to  the  XOR  mapping  problem  with  10  parent  lu  lwoiks  f  or  lu  twoik  .  that  were  !ull\ 
connected  from  the  on.set,  this  .search  technique  quickly  found  a  fully  conruvied  J  1  .tnn  tiire  and  vseiglit  .id.ipialii in 
fxxurred  over  subsequent  generations,  fo  make  the  problem  more  challcngnig,  ca  li  lu  twnik  w.is  insl.inli.iti  d  wi(h  ,i  >•>'  ' 
probability  of  any  given  connection  being  pre.sent,  ITie  resulting  sliucture  is  shown  in  i  ig  ft  and  tiu  sl.iti  tu  .d  li.i!.)  i 
showm  in  Fig.  15  This  experiment  was  conducted  with  weighting  eofttieienis  ot  o  /  (!  O  OOOl ,  y  O.iMtl  1  he  hidden 
layer  domain  consistetl  ot  50  hidden  units  I'he  .statistics  at  the  end  ol  lOOO  gcm  ialn 'iis  an  ,V  /  f 

E  -  0.000000,  J  O.0I23(X)  and  a  dilution  ratio  of  D  0. 1 1 


Figure  11.  Evolving  weights  and  structure  for  the  XOR  mapping:  (a)  cost  /unction,  (b)  mean  sum 
squared  pattern  error,  (c)  number  of  connections,  and  (d)  the  number  of  neurons. 


Figure  12.  Evolving  weights  and  structure  for  .t-bit  parity:  (a)  objective  function,  (b)  th 
squared  pattern  error,  (c)  number  of  connections  and  frf)  number  of  neurons 


Figure  13.  The  final  evolved  architecture  for 
the  3-bit  parity  mapping  problem. 


Figure  14.  The  final  evolved  architecture  for  the  XOK 
mapping  problem. 


Figure  15.  FJvolving  weights  and  structure  for  the  XOR  mapping:  (a)  co.st  function,  (h)  mean  sum 
squared  pattern  error,  (c)  number  of  connections,  and  (d)  the  number  of  neurons. 


6.  CONCLUSIONS 


Experiments  have  shown  that  evolutionary  programming  can  be  used  for  simultancriusly  determining  both  network 
architecture  and  parameters.  These  results  indicate  that  the  EP  search  paradigm  has  high  potential  for  automating  the  design 
of  neural  networks.  It  has  been  demonstrated  that  the  general  architecture  shown  in  Fig.  1  can  be  achieved  for  simple  binary 
mapping  problems.  However,  some  applications  may  be  hindered  by  the  excess  memory  rcr^uireinents  needed  by  multi-agent 
searches'^  or  the  training  time  necessary  to  obtain  adequate  convergence.  If  the  design  prr>ccss  for  a  .specific  problem  is  not 
capable  of  being  fully  automated,  then  this  technique  may  prove  mseful  in  providing  a  designer  with  insight  to  the  task  at  hand 
as  well  as  a  reasonable  starting  point  for  further  investigations. 

Caution  should  be  used  in  trying  to  obtain  minimal  .size  networks.  As  Sietsina  and  Dow’-“  point  out,  pruning  networks 
reducsis  their  generalization  capabilities  if  noise  is  present.  Experience  has  that  shown  that  adding  a  slight  amount  of  noise 
to  the  training  data  will  result  in  more  robust  classifiers.  Since  only  binary  mapping  problems  were  investigate*!,  it  u  not 
clear  how  the  approach  given  in  this  .-.tudy  will  work  on  classification  or  continuous  mapping  problems.  This  remains  an 
area  of  further  research. 

Nevertheless,  stochastic  training  techniques  are  becoming  p.evalent  in  neuriKomputiiig  (especially  in  hardware 
implementations^').  Further  research  should  allow  for  a  more  general  connectivity  .structure  including  interconnections  amcmg 
the  hidden  units  as  well  as  connections  between  input  and  output  neurons.  Since  model  structure  is  continually  being 
modified,  it  is  incumbent  that  an  information  criteria  /  be  addre.s.sed  either  expie  .t!y  or  implicit  in  the  objective  function  (r. 

S'J 

J  -  aE  *  /3/V '  UU) 

providing  a  viable  approach  for  automated  model  selection  based  not  only  on  error  and  computation  cost,  but  on  the 
information  content  as  well. 
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